Streaming mode is not supported at the moment

Testing the XllmForCausalLM support

This PR (with the xllm commit) was verified end-to-end against a real xllm 8B checkpoint.

1. Install

git clone --branch tgat/multi-format-tool-parser-v0.12.0 https://github.com/LLM360/vllm.git
cd vllm
uv venv --python 3.12 .venv
VLLM_USE_PRECOMPILED=1 uv pip install --editable .

2. Serve a checkpoint with model_type: "xllm"

.venv/bin/vllm serve /path/to/xllm/checkpoint \
    --trust-remote-code \
    --tensor-parallel-size 8 \
    --port 8000 \
    --reasoning-parser k2_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser multi_format

You should see in the logs:

INFO [model.py:637] Resolved architecture: XllmForCausalLM
INFO [gpu_model_runner.py:3549] Model loading took 2.1271 GiB memory and 46.158572 seconds
INFO: Application startup complete.

3. Sanity check — simple chat

import requests
resp = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "/path/to/xllm/checkpoint",
    "messages": [{"role": "user", "content": "What is 15 * 23?"}],
    "max_tokens": 256,
    "temperature": 0.7,
    "top_p": 0.95,
    "extra_body": {"chat_template_kwargs": {"reasoning_effort": "high"}},
})
print(resp.json()["choices"][0]["message"])

Expected: a valid response with reasoning_content and content populated (e.g. "15 × 23 = 345.").

4. Verify reasoning + tool parsers

Reasoning effort ↔ think token mapping:

• reasoning_effort=high → <think>
• reasoning_effort=medium → <think_fast>
• reasoning_effort=low → <think_faster>

The reasoning_content field should be populated on the response when the model emits a think block.

Tool calling (default format):

resp = requests.post("http://localhost:8000/v1/chat/completions", json={
    "model": "/path/to/xllm/checkpoint",
    "messages": [{"role": "user", "content": "What's the weather in Tokyo?"}],
    "tools": [{"type": "function", "function": {
        "name": "get_weather",
        "description": "Get current weather",
        "parameters": {"type": "object",
                       "properties": {"location": {"type": "string"}},
                       "required": ["location"]}
    }}],
    "tool_choice": "auto",
})
# Expected: choices[0].message.tool_calls[0].function.name == "get_weather"
#           arguments parses to {"location": "Tokyo"}

5. End-to-end Harbor task (optional)

export DAYTONA_API_KEY=<key>
export OPENAI_API_KEY=dummy
harbor run \
  -p /path/to/LiteCoder/build-go-hello-binary \
  -e daytona -a terminus-2 \
  -m "openai//path/to/xllm/checkpoint" \
  --ak api_base=http://localhost:8000/v1 \
  --ak reasoning_effort=medium \
  -n 1

Expected: reward = 1.0 — model clones a Go repo, builds the binary, writes output to /app/output.txt.

What the changes actually do

• vllm/model_executor/models/xllm.py: new model implementation. Structurally identical to Llama (same weight names) but replaces RMSNorm with GroupRMSNorm, which computes variance per group of hidden_size/n_groups dims instead of the full hidden dimension. Required because xllm checkpoints are trained with layernorm_num_groups=4 — using standard RMSNorm would produce numerically wrong outputs.
• vllm/model_executor/models/registry.py: one-line addition to map "XllmForCausalLM" → xllm module.

Without these changes, vLLM fails with "unsupported architecture: XllmForCausalLM" at startup.

✅ Test results on xllm 8B checkpoint

Ran the test plan from the previous comment against a real xllm 8B checkpoint (config.json architecture = XllmForCausalLM, layernorm_num_groups=4). Server brought up with:

.venv/bin/vllm serve /path/to/xllm/checkpoint \
    --trust-remote-code \
    --tensor-parallel-size 8 \
    --port 8000 \
    --reasoning-parser k2_v3 \
    --enable-auto-tool-choice \
    --tool-call-parser multi_format

Logs confirm architecture resolution:

INFO [model.py:637] Resolved architecture: XllmForCausalLM
INFO [gpu_model_runner.py:3549] Model loading took 2.1271 GiB memory and 46.158572 seconds
INFO: Application startup complete.

Parser + model test suite: 14/15 passed

Math answers being correct is the key signal that GroupRMSNorm (per-group variance with n_groups=4) is mathematically correct — using standard RMSNorm would have produced garbled outputs.

What's verified by this run

1. XllmForCausalLM resolves via the registry addition.
2. Weight loading works — same weight names as Llama, 36 shards loaded, 2.1 GiB/GPU across TP=8.
3. GroupRMSNorm computes correctly (numeric answers are right, reasoning is coherent).
4. Both the k2_v3 reasoning parser and multi_format tool parser work correctly on top of the new model.

End-to-end sanity: Harbor task

Also ran a full Harbor LiteCoder/build-go-hello-binary trial against this checkpoint with the terminus-2 agent pointed at the local vLLM server (api_base=http://localhost:8000/v1). The model successfully cloned the Go repo, built the binary, and wrote the output file — reward = 1.0 in a ~1m44s run. So the changes hold up under a real multi-turn agent loop, not just single-shot requests.